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DATA COMMUNICATION NETWORK 

5 CROSS-REFERENCE TO RELATED APPLICATIONS 

[0001] This application is related to and claims priority to United Kingdom of 

Great Britain Patent Application Serial No. GB 0306855.8, filed March 25, 2003, 
inventors Kevin Baughan, Constantinos Christofi Constantinou, Alexander Sergeevich 

1 0 Stepanenko, and Theodoros Arvanitis, entitled "Data Communication Network", which is 
commonly assigned herewith, the contents of which are incorporated herein by reference, 
and with priority claimed for all commonly disclosed subject matter, and further is related 
and claims priority to PCT International Patent Application Serial PCT/EP2004/050195, 
filed February 24, 2004, inventors Kevin Baughan, Constantinos Christofi Constantinou, 

1 5 Alexander Sergeevich Stepanenko, and Theodoros Arvanitis, entitled "Topology 
Aggregation For Hierarchical Routing", which is commonly assigned herewith, the 
contents of which are incorporated herein by reference, and with priority claimed for all 
commonly disclosed subject matter. 

20 BACKGROUND OF THE INVENTION 

Field of the Invention 

[0002] This invention relates to a data communication network. 

25 

Description of the Related Art 

[0003] A simple data communication network is illustrated in Figure 1 of the 

accompanying drawings. The network comprises a plurality of nodes N0.1 to NO. 10 
30 interconnected by transmission lines LI to LI 3, normally referred to simply as links. The 
nodes will generally comprise some form of intelligent switching device such as a frame 
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switch, a packet switch, a cell switch or a label switch. The transmission lines may 
comprise wire links, fibre optic links, infrared links, wireless links or combinations of 
these. The particular nature of the transmission lines is not important: they are simply a 
means of enabling data to be transported between the nodes. As illustrated in Figure 2, 
5 most of the nodes and their associated transmission lines form closed rings Rl . 1 to Rl .5. 
The nodes N0.1, N0.2 and NO. 3 and their associated transmission lines LI and L2 are 
connected in a tree structure Tl.l with node N0.1 at the remote end of the tree. Routing 
data across the tree structure Tl .1 is deterministic as there is only one path to the 
destination that the data can follow. The present invention, allows for the rest of the 
1 0 network to be reduced to a logical tree structure, through a series of abstractions, so that 
together with the physical tree structure Tl.l, deterministic routing can be achieved 
across the whole network. 

[0004] Any of the nodes NO. 1 to NO. 1 0 can be used to connect to a host machine 

1 5 or machines (not shown), possibly themselves interconnected by means of a LAN (local 
area network). 

[0005] As is well known, in such networks data to be transmitted is segmented, 

and each segment is encapsulated in a packet, frame or cell (depending on the protocol 

20 being used) for transfer across the network. Each packet, frame or cell will contain, as 
well as the data to be transmitted, control data, which will generally comprise such 
information as the source and destination address and may also include such information 
as a circuit identifier or label that corresponds to a connection between two nodes in the 
network. Various different methods are used for routing the packets, frames or cells 

25 across the network. Generally speaking the switching device at each node includes a 
storage device, which stores a routing table, which is accessed according to the control 
data contained within the packet, frame or cell in order to forward the packet, frame or 
cell to its destination by a preferred path. The manner in which the table is populated 
and acted upon gives rise to the different methods of routing. The software associated 

30 with populating these tables is controlled by a routing algorithm. The primary purpose of 
the routing algorithm, of which there are several in common use, is to determine the 
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preferred path to the destination, based on one or more parameters, and to populate the 
routing table in each node to give effect to the result of this determination. Then, when a 
packet, frame or cell arrives at a particular node, the contents of the table will dictate the 
next hop of the path upon which it is transmitted from the node. Although the present 
5 invention is connected with routing, it is in fact transparent to the particular method of 
switching that is used. 

[0006] Thus it will be seen that where each node contains routing information, the 

routing information is used to forward the packet, frame or cell on a specific path to its 

1 0 destination. The type of information stored will depend upon the sophistication of the 
network and the particular routing method that is used. The routing information in each 
node may also take into account the performance of the network - for example whether 
there are any congestion problems or link failures ahead. In order to keep the routing 
information at each node up-to-date, it is normally provided that the information is 

1 5 updated periodically to cater for any changes in the network. This update may be 

achieved through packets, frames or cells that are purely control messages. Although the 
present invention is connected with the use of performance information updates, it is in 
fact transparent to the particular method of update that is used. 

20 SUMMARY OF THE INVENTION 

[0007] The present invention is directed to what is considered to be a new way of 

looking at networks, by basing the routing algorithms, which in turn control the content 
of the information in the routing tables, on a recursive abstraction of the physical network 
25 into a series of logical levels. 

[0008] According to a first aspect of the present invention, there is provided a 

method of generating a routing table of destinations for a first physical node of a data 
communication network which network consists of a plurality of nodes, links 
30 interconnecting said nodes and a plurality of destinations associated with respective 
nodes, comprising the steps of: 
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a) collecting topological information on at least a part of the data 
communication network in terms of physical nodes and links between physical nodes; 

b) embedding the collected topological information in a plane 
corresponding to a first network level; 

5 c) identifying one or more closed loops of interconnected nodes lying in 

the plane of said network level; 

d) for a first further network level, assigning a virtual node for each 
closed loop of interconnected nodes in the previous network level, each virtual node 
being representative at the further network level of the nodes of the corresponding closed 

1 0 loop in the previous network level and any destinations associated with those nodes; 

e) identifying links between said virtual nodes, the links corresponding to 
nodes in the previous network level that are common to two or more virtual nodes in the 
further network level; 

whereby the route between said first physical node and a destination 
1 5 associated with a further physical node of the data communication network is defined for 
each further physical node in relation to a network level at which said first physical node 
and the further physical node are interconnected by a single path; and 

f) populating the routing table of the first physical node for each 
destination with the set of paths that belong to the previous network level corresponding 

20 to the single path at the network level at which the first physical node and said destination 
are interconnected. 

[0009] Ideally, the collected topological information is used to generate a 

subnetwork and the subnetwork is embedded in the plane corresponding to the first 
25 network level to produce a planar embedded graph from which faces are identified 
corresponding to the closed loops. 

[0010] Moreover, the above steps may be repeated for additional further network 

levels. 

30 It will, of course, be apparent that whilst it is necessary, in accordance 

with the present invention for the first physical node and the further physical node to be 



-4- 



U.S. Express Mail No. EQ 139 078 383 US 
Substitute Specification as Amended 

interconnected by a single path, for the purposes of populating the routing table it may 
not be necessary, in all cases, to continue the recursive abstraction to the level at which a 
virtual node is allocated that is representative of the single path between the first and 
further physical nodes. Instead identification of the set of paths between the first and 
5 further nodes at the previous network level may be all that is required. 

[0011] Destination address information may be collected together with the 

topological information or may be advertised across the data communication network 
separately. 

10 

[0012] According to a second aspect of the present invention, there is provided a 

network node suitable for use in a data communication network which network consists 
of a plurality of nodes, links interconnecting said nodes and a plurality of destinations 
associated with respective nodes, the network node comprising: 
15 an input/output interface for data input to and output from the network 

node; 

data storage adapted to store a routing table; 
a processor for populating said routing table; 

a selector for selecting a path across said data communication network to a 
20 destination on the basis of information contained in said routing table; and 

program storage means in which is stored a set of instructions for 
populating said routing table, the set of instructions comprising instructions for: 
a) collecting topological information on at least a part of the data 
communication network in terms of physical nodes and links between physical nodes; 
25 b) embedding the collected topological information in a plane 

corresponding to a first network level; 

c) identifying one or more closed loops of interconnected nodes lying in 
the plane of said network level; 

d) for a first further network level, assigning a virtual node for each 
30 closed loop of interconnected nodes in the previous network level, each virtual node 
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being representative at the further network level of the nodes of the corresponding closed 
loop in the previous network level and any destinations associated with those nodes; 

e) identifying links between said virtual nodes, the links corresponding to 
nodes in the previous network level that are common to two or more virtual nodes in the 
5 further network level; 

whereby the route between said first physical node and a destination 
associated with a further physical node of the data communication network is defined in 
relation to a network level at which said first physical node and the further physical node 
are interconnected by a single path; and 
10 f) populating the routing table of the first physical node for each 

destination with the set of paths that belong to the previous network level corresponding 
to the single path at the network level at which the first physical node and said destination 
are interconnected. 

1 5 [0013] The present invention is also directed to a network of such nodes. 

[0014] In a third aspect the present invention provides a method of operating a 

network node as described above in a data communication network, the method 
comprising the steps of: when data to be transmitted to. a destination on the data 

20 communication network is input to the network node, the selector accesses the routing 
table to identify the route for the required node associated with the destination of the 
data; where the required node is linked at a network level to the network node by a single 
path, the selector determines a direction of circulation of the data around the underlying 
closed loops at each previous level in which the network node participates in order to 

25 achieve deterministic routing of the data across the network. 

[0015] Ideally, a path for the destination of input data is adaptively selected with 

respect to a closed loop at a particular network level, based on available information on 
the network state at that level. 

30 
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[0016] Moreover, address and network performance information may be 

distributed at each network level with the nodes themselves as the destinations. 

[0017] Within each closed ring at every level in the architecture, each node may 

5 exchange address and performance information with the other node or nodes in the closed 
ring. Typically, the address and performance information at level n is a summary of the 
corresponding information at the preceding network level n-1. The process of distributing 
address and performance information across the network is referred to as advertising and 
examples of how it can be achieved are described in detail below. 

10 

[0018] The invention, unlike other methods in common use that seek to determine 

for each route a unique or limited number of paths based on optimising a particular set of 
one or more parameters, is able to retain a rich set of paths for each route and to exploit 
them at each logical level as appropriate, based on the performance information available. 

1 5 This routing decision is made in accordance with the routing information stored at each 
node and this routing information is created by running a routing algorithm which is 
based on a recursive abstraction of the network into logical levels, as described above. 
When a packet, frame or cell arrives at, or is originated at, a node, the node is expected to 
forward the packet, frame or cell onwards through an appropriate output port in 

20 accordance with the information contained in the routing table. This process is repeated 
at each node as the packet, frame or cell travels across the network. 

[0019] The choice of paths that may be utilized at each node in order to achieve 

deterministic routing across the network is specified by a routing algorithm that is based 

25 on a recursive abstraction of the network into logical levels. The selection of a specific 
path from the choice of paths being made at each node may be based on the currently 
available performance information. The node then forwards the packet, frame or cell 
onwards through the output port that corresponds to the specific path that has been 
selected. The overall process of successively forwarding data node-by-node across the 

30 network is referred to as routing and examples of how it can be achieved are described in 
detail below. 
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[0020] The concepts described herein are independent of the particular method of 

switching used, of the particular method of advertising used, of the particular type of 
addressing used, and of the particular type of performance information used. Examples 
of particular techniques for advertising and routing as well as a number of other issues 
5 associated with the correct functioning of the network are described below. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0021] In order that the invention may be better understood, several embodiments 

1 0 thereof will now be described by way of example only and with reference to the 
accompanying drawings in which: 

[0022] Figure 1 is a diagrammatic view of a data communication network; 

1 5 [0023] Figure 2 is a diagram to show how the network of Figure 1 may be divided 

into closed rings; 

[0024] Figure 3 is a block diagram of a node suitable for use in a network in 

accordance with the present invention; 

20 

[0025] Figure 4 is a diagram of an example physical data communication network 

used to explain the invention in detail; 

[0026] Figures 5 and 6 diagrammatically show recursive abstraction of the 

25 physical network of Figure 4 in accordance with the present invention; 

[0027] Figures 7, 8, 9 and 10 diagrammatically show circulation vectors at each 

level of abstraction of the physical network of Figure 4 in accordance with the present 
invention; 

30 
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[0028] Figure 1 1 diagrammatically shows the forwarding of data packets across 

the network of Figure 4 in accordance with the present invention; 

[0029] Figures 12, 13 and 14 diagrammatically show the backtracking of data 

5 packets across the network of Figure 4 in accordance with the present invention; 

[0030] Figures 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26 and 27 

diagrammatically show the dissemination of address and performance information across 
the network of Figure 4 in accordance with the present invention; 

10 

[0031] Figures 28, 29, 30, 31 and 32 diagrammatically show the transmission of 

data across the network of Figure 4 without the use of explicit circulation vector 
information in accordance with the present invention; 

1 5 [0032] Figures 33 and 34 are diagrams showing how the physical network of 

Figure 1 may be recursively abstracted into three logical levels, according to the present 
invention, in order to form a loop-free structure that will enable deterministic routing. 

[0033] Figure 35 is a diagram showing how early termination of the process to 

20 recursively abstract the network of Figure 4 into a series of logical levels can be used to 
impose a hierarchical network architecture in accordance with the present invention; and 

[0034] Figures 36, 37, 38, 39, 40 and 41 are diagrams showing how non-planar 

links may be successfully incorporated into the recursive abstraction of the network into a 
25 series of logical levels in accordance with the present invention. 

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS 

[0035] Referring firstly to Figure 3, there is shown a block diagram of a node 30 

30 suitable for use in the communication of data across a network of such nodes. The node 
30 comprises 3 logical input/output ports which map onto 3 input/output interfaces 3 1 , 



-9- 



U.S. Express Mail No. EQ 139 078 383 US 
Substitute Specification as Amended 

32, 33 each of which connects via a respective queuing memory 34, 35, 36 to a selector in 
the form of a switching fabric 37. Although shown with 3 input/output ports, it will be 
understood that the node may in practice contain any number of ports, from two upwards. 
In some situations, for example wireless transmission, multiple logical ports can be 
5 mapped onto a single physical interface. Each interface, in this example, is connected via 
a transmission line to an adjacent node in the network, or may be connected by a 
transmission line to a host, which may be a stand-alone computer, or one of multiple 
computers connected via a LAN. 

1 0 [0036] The switching fabric 37 receives packets, frames or cells from one of 

interfaces 31, 32 or 33 and forwards them to another of the interfaces 31, 32 or 33 for 
onward transmission. The memories 34, 35 and 36, which may be in the form of buffers, 
provide a queuing facility in the event that the node receives more data within a particular 
time frame than it can handle. 

15 

[0037] The decision as to which interface the incoming data is forwarded by the 

switching fabric 37 is taken on the basis of routing information contained in a routing 
table 38 which is maintained by a processor and associated circuitry 39. The processor 
39 is linked to the table 38 by an input/output interface 40. Associated with the 

20 processor 39 is a memory 41, which contains the software, which controls the processor, 
and may also contain a copy of the current contents of the routing table. The software is 
based on intelligent algorithms which essentially comprise a set of rules which dictate 
how any particular packet, frame or cell received by or originated at the node is to be 
forwarded, bearing in mind such factors as its required destination address, any label or 

25 circuit identification that is present and the state of the network. The processor also 
updates the information contained in the routing table on the basis of network status 
information advertised across the network. 

[0038] All packets, frames or cells are examined on arrival to ascertain whether 

30 they are control messages which contain purely control information - for example, 

changes in address and performance information which may be used to update the routing 
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table. Any such packets, frames or cells are forwarded by the switching fabric 37 to the 
processor 39 via a queuing memory 42 and input/output interface 43. These control 
messages are used to set up the node in the first place, and to keep it up to date with any 
changes in the network, which require changes in the routing table, or indeed in the 
5 algorithms themselves. 

[0039] Reference is now made to Figures 4 to 32, which illustrate the manner in 

which a physical network may be recursively abstracted into a series of logical levels and 
consider the processes of advertising and routing in connection with this recursive 
1 0 abstraction. 

[0040] Before discussing a typical real life network example, such as that shown 

in Figure 1, it is easier to explain the overall principles of the routing technology of the 
invention by considering the example network of 12 nodes as shown in Figure 4. 

15 

[0041] The first level of abstraction is the planar physical network of Figure 4. In 

Figure 5, we identify the minimal simple cycles, as defined in graph theory, of Level 1. 
As can be clearly seen from Figure 5, these cycles are minimum length closed rings, 
measured in numbers of hops, or links, and excluding nodes that are purely transit. Non- 
20 planar links are dealt with in a later example below. Level 2, which is a virtual level 
(otherwise referred to herein as a "logical level"), is derived from level 1 by defining a 
level 2 virtual node corresponding to each level 1 ring. First, all tree structures of level 1 
are collapsed into the nodes they attach to, as routing over them is deterministic. Then a 
level 2 link is drawn between level 2 nodes, provided their corresponding level 1 rings 
25 share at least a pair of level 1 nodes, unless this partitions the network into disjoint sub- 
networks. In this case, two level 2 nodes can be joined together by a level 2 link, 
exceptionally, if the corresponding rings have a single level 1 node in common (a cut 
node in graph theory). The procedure can be iterated as many times as required up to 
whichever level renders the graph of the network ring-free, as is shown in Figure 6 (a)-(d) 
30 and elaborated below. It should be noted that a single link whose failure partitions the 
network (a cut-edge in graph theory), must be treated as a degenerate loop. 
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[0042] Ring RL 1 at Level 1 in Figure 6(a) becomes node N 1 . 1 at Level 2 in 

Figure 6(b). Similarly, Rl .2 becomes Nl .2 and Rl .5 becomes Nl .5. A pair of nodes, 
such as Nl.l and N1.2 at Level 2 in Figure 6(b), is considered to be directly connected at 
5 Level 2, as Rl . 1 and Rl .2 share Level 1 nodes N0.2 and NO. 10 together with their 
interconnecting link at Level 1 . Similar considerations show that Nl .5 and Nl . 1 are 
linked together at Level 2, as well as node N1.2 and N1.5. Nodes Nl.l, N1.2 and N1.5 
now form a minimal closed ring at Level 2, which is denoted by R2.1 in Figure 6(b). 
This ring R2.1 at Level 2 in Figure 6(b) becomes node N2.1 in Figure 6(c). Similarly, 
10 R2.2 becomes N2.2, R2.3 becomes N2.3 and R2. 4, becomes N2.4. In a similar manner, 
these now form a minimal closed ring at Level 3, which is denoted by R3.1 in Figure 
6(c). This ring R3.1 at Level 3 in Figure 6(c), becomes node N4.1 in Figure 6(d). As the 
latter is a ring-free network, the process terminates and the abstraction is complete. 

1 5 [0043] In going from Figure 6(a) to 6(b), a process has been performed which is 

defined as logical network abridgement. 

[0044] Two example scenarios are now considered of how advertising and 

routing may be achieved across a data communication network which utilises routing 
20 algorithms based on logical network abridgement as described above. 

[0045] In the first example scenario, conventional link-state routing 

protocols can be used to obtain information on the topology of a network, which can then 
be recursively abstracted into a series of logical levels. Conventional link-state routing 

25 protocols can also be used to advertise address information that together with the 

abstracted logical levels are used to enter destinations and their associated set of paths 
into the routing table as described below. Conventional link-state routing protocols can 
also be used as described later to advertise performance information in order to enable 
optimisation of the traffic distribution across the alternative paths that make up each 

30 route. Data can then be routed across the network by dynamically linking together 
connections that correspond to the paths across the logical levels derived from the 
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recursive abstraction of the network. An example of how this might be implemented 
would be through using the destination address information in an IP packet together with 
path information present in one or more multi-protocol label switch headers, or one or 
more Ethernet VLAN headers. 

5 

[0046] As an existing link-state routing protocol has been used to communicate 

both topology and address information across the network, each node is able to construct 
a complete view of the physical topology of the network and the addresses of the 
destinations associated with each node in the physical network as shown in Figure 4. 

10 

[0047] The logical network abridgement process can then be used to establish the 

recursive abstraction of the physical network into a series of logical levels as shown in 
Figure 6. For each level, prior to reaching a loop-free structure, it is possible to identify 
for each logical node a pair of connections to each of the other logical nodes that are 

1 5 members of the same ring. These connections are termed circulation vectors as they 
represent the different orientations of circulation that are possible when travelling 
towards a specific destination node on a specific ring. The vectors are defined by their 
circulation and destination. They are denoted by the name of the ring, together with the 
direction of circulation that is represented by either a "+" sign (positive rotation) or a 

20 sign (negative rotation) and the destination node. The choice of defining which direction 
of rotation is positive is arbitrary, but whatever is chosen must be applied consistently for 
all of the rings at all of the levels. The opposite direction of rotation to positive is defined 
as being negative. In the attached Figures, positive rotation is defined as being clockwise 
and negative therefore as anticlockwise. 

25 

[0048] Figure 7 shows the circulation vectors at level 1 that node NO. 1 can use to 

reach nodes N0.2, N0.8, N0.9 and NO. 10 on ring Rl.l. Figure 8 shows the circulation 
vectors at level 2 that node Nl . 1 can use to reach nodes Nl .2 and Nl .5 on ring R2. 1 . 
Figure 9 shows the circulation vectors at level 3 that node N2.1 can use to reach nodes 
30 N2.2, N2.3 and N2.4 on ring R3.1. 
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[0049] Each ring at each level can be correlated with the addresses of destinations 

that are reachable on that ring by reversing the abstraction process and identifying the 
physical nodes that are associated with that part of the abstracted network. It can 
therefore be seen that as you move up the levels increasing proportions of the addresses 
5 on the network are reachable, until at the highest level of abstraction all addresses on the 
network are reachable. 

[0050] Figure 10 shows the view of the circulation vectors at each level of the 

abstracted network from the perspective of node N0.1 . By reversing the abstraction 

1 0 process as described earlier, it can be seen that: the circulation vectors at level 1 on ring 
Rl.l can reach the addresses associated with nodes N0.2, NO. 8 and NO. 9 and NO. 10; the 
circulation vectors at level 2 on rings R2.1 and R2.4 can also reach the addresses 
associated with all the other nodes in the network except node NO. 5; and that the 
circulation vectors at level 3 on ring R3.1 can also reach the addresses associated with 

15 nodeN0.5. 

[0051] The virtual node corresponding to the closed ring at the lowest level that 

the destination address is first reachable therefore describes the route to the destination. 
The choice of circulation vectors on the corresponding closed ring, together with all the 
20 circulation vectors that belong to lower levels within that part of the abstracted network, 
represent the path diversity that is possible in following this route. 

[0052] Performance information describing the status of each ring at each level 

can then be used to make informed choices of how best to distribute the offered traffic 

25 against the available resources in the network. Performance information concerning each 
ring is therefore advertised to each member of a ring for each ring at all levels in the 
abstracted network. One example of how this can be achieved is through an enhancement 
to the traditional flooding mechanism used in link-state routing protocols. The 
enhancement is termed restricted flooding. In restricted flooding, each piece of 

30 performance information is associated with a specific ring at a specific level. The 

performance information is flooded through the network until it reaches a node that is not 
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a member of the ring with which the performance information is associated. This node 
simply discards the performance information specific to the ring of which the node is not 
a member, but continues to flood any other relevant performance information to its 
neighbours. 

5 

[0053] Thus, in order to populate the routing table of each individual network 

node topological information is first collected and then used to form a subnetwork which 
is embedded in a plane to form a planar embedded graph from which faces can be 
identified corresponding to the closed loops referred to earlier. The analysis of the 

1 0 network proceeds in the manner described above generating increasingly higher network 
levels until preferably a wholly deterministic path to all points of the network is 
identified. This information is then used to populate the routing table by identifying for 
each potential data destination its associated physical node and the network level at 
which that physical node is interconnected by a single path with the particular network 

1 5 node whose routing table is being populated. This single path is entered into the routing 
table as the route to that destination, and the set of circulation vectors around the 
underlying closed loops at each previous level in which the network node participates are 
entered into the routing table as the alternative paths that can be used in order to follow 
this route. 

20 

[0054] As mentioned earlier the information contained in the routing table can be 

updated by the processor in response to network status information advertised over the 
network in order to optimise the distribution of traffic over the available paths. 

25 [0055] Reference is now made to Figure 1 1 . This considers the example of data 

being routed across the network. The source of the data is a host machine attached to 
node N0.1 and the destination is a host machine with address xyz attached to node N0.6. 
As a general rule, data is always routed at the lowest level that is operational. A higher 
level path is only used if there are no paths operational at a lower level. The lowest level 

30 at which node N0.1 is aware of destination xyz is a route based on node N2.4. The level 
2 paths to destination address xyz are therefore defined by the circulation vector R2.4+ to 
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Nl .4 and R2.4- to Nl .4. Based on level 2 performance information, node NO. 1 selects 
R2.4+ to Nl .4. Node N0.1 therefore adds a header to the data. The header identifies the 
circulation vector R2.4+ to N1.4. Node N0.1 also knows that this level 2 circulation 
vector R2.4+ requires the data to be sent from Nl . 1 to Nl .5. NO. 1 also knows that the 
5 link between Nl . 1 and Nl .5 is the pair of gateway nodes N0.9 and NO. 10. The level 1 
paths to follow R2.4+ are therefore defined by the circulation vector R1.1+ to NO. 10 and 
Rl.l- to N0.9. Based on level 1 performance information, node N0.1 selects R1.1+ to 
NO. 10. Node N0.1 therefore adds a further header to the data identifying the circulation 
vector R1.1+ to NO. 10. Node N0.1 also knows that to leave on Rl.l + it must send the 
1 0 data on the link to node N0.2. The forwarding decision has therefore been reached and 
node N0.1 sends the data complete with the header information to node NO. 2. 

[0056] As a general rule, nodes must always maintain the direction of circulation 

until the data reaches the destinations specified by all circulation vectors that are present. 
15 So when node NO. 2 receives the data it looks at the outer header indicating that it is 

travelling on Rl.l + to NO. 10. Node N0.2 knows that it is not the destination and that to 
leave on Rl . 1+ it must send the data on the link to node NO. 10. The forwarding decision 
has therefore been reached and node N0.2 sends the data complete with the existing 
header information to node NO. 10. 

20 

[0057] When node NO. 1 0 receives the data it looks at the outer header indicating 

that it is travelling on Rl .1+ to NO. 10. Node NO. 10 knows it is the destination at level 1 
and therefore removes the outer header and looks at the next header indicating that it is 
travelling on R2.4+ to N1.4. Node NO. 10 knows that it is not the level 2 destination and 

25 that to leave on R2.4+ it must send the data on the link from N 1.5 to N1.4. NO. 10 also 
knows that the link from Nl .5 to Nl .4 is the pair of gateway nodes N0.9 and NO. 12. The 
level 1 paths to follow R2.4+ are therefore defined by the circulation vector Rl .5+ to 
NO. 12 and R1.5- to N0.9. Based on the level 1 performance information, node NO. 10 
selects R1.5+ to NO. 12. NO. 10 therefore adds a new outer header to the data packet 

30 identifying the circulation vector Rl .5+ to NO. 12. Node NO. 1 0 also knows that for the 
data to leave on Rl .5+ the data must be sent on the link to node NO. 1 1 . The forwarding 
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decision has therefore been reached and node NO. 10 sends the data complete with the 
header information to node NO. 1 1 . 



[0058] When node N0.1 1 receives the data it looks at the outer header indicating 

5 that it is travelling on Rl .5+ to NO. 12. Node NO. 1 1 knows that it is not the destination 
and that to leave on Rl .5+ it must send the data on the link to node NO. 12. The 
forwarding decision has been reached and node N0.1 1 sends the data complete with the 
existing header information to node NO. 12. 

1 0 [0059] When node NO. 1 2 receives the data it looks at the outer header indicating 

that it is travelling on R1.5+ to NO. 12. Node NO. 12 knows that it is the destination at 
level 1 and therefore removes the outer header and looks at the next header indicating 
that it is travelling on R2.4+ to Nl .4. Node NO. 12 knows that it is also the level 2 
destination address and therefore also removes this header. As there is no further header, 

1 5 node NO. 1 2 treats the data as if it had originated locally and considers the destination 
address xyz. The lowest level that node NO. 12 is aware of destination xyz is the route 
based on node Nl .4. The level 1 paths to destination address xyz are therefore defined by 
the circulation vectors R1.4+ to N0.6 and R1.4- to N0.6. Based on level 1 performance 
information, node NO. 12 selects R1.4+ to N0.6. Node NO. 12 therefore adds a header to 

20 the data identifying the circulation vector Rl .4+ to N0.6. Node NO. 1 2 also knows that to 
leave on Rl .4+ it must send the data on the link to node N0.6. The forwarding decision 
has therefore been reached and node NO. 12 sends the data complete with the header 
information to node N0.6. 



25 [0060] When node NO. 6 receives the data it looks at the outer header indicating 

that it is travelling on Rl .4+ to N0.6. Node N0.6 knows it is the destination at level 1 and 
therefore removes the header. As there is no further header, node NO. 6 treats the data as 
if it had originated locally and considers the destination address xyz. Node NO. 6 
recognises that it is the address of a locally attached host and delivers the data. 

30 
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[0061] As with existing multi-protocol label switching solutions, it is also 

possible to remove a circulation vector, corresponding to a particular level, at the 
penultimate node at that level. This is the because the next node at that level will be the 
destination node, and its first action will be to recognise that it is the destination at that 
5 level and remove the circulation vector. This can reduce the circulation vector overhead 
required to manage the flow of data on the network. However, circulation vector 
information should not be removed if the network is to support backtracking. 

[0062] Reference is now made to Figure 12. Backtracking allows data to 

1 0 backtrack and find a new path through the network should it reach a point in the network 
where there is no possibility of progressing towards the destination on its current set of 
circulation vectors. For example, in the previous routing scenario, if upon arrival at 
N0.1 1 it was found unexpectedly, for example as a result of recent damage, that the data 
could not be forwarded on Rl .5+ to NO. 12, the data could be backtracked and forwarded 
15 onR1.5-toN0.12. 

[0063] Reference is now made to Figure 13. This shows that if the origin 

information associated with each circulation vector is kept, the data can be backtracked to 
the starting point of the circulation vector and a fresh decision on the best path to be 

20 followed made based on updated performance information. For example, in the previous 
scenario, if upon arrival at N0.1 1 it was found unexpectedly that the data could not be 
forwarded on Rl .5+ to NO. 1 2, the data could be backtracked to NO. 1 0 the origin of the 
last level 1 circulation vector. Node NO. 10 could then decide to follow instead R1.5- to 
N0.9 and from node N0.9 it could reach node N0.6 either way around ring Rl .4 (Rl .4+ 

25 shown as selected). 

[0064] Reference is now made to Figure 14. This shows that if for some reason 

no paths are available after backtracking along a level n circulation vector and the node 
still has higher level circulation vectors, the process can be recursively repeated and the 
30 data can be backtracked even further using the level n+1 circulation vector - until a path 
can be found or the node associated with the source address is reached. For example, in 
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the situation above, if there was no path on R1.5- to NQ.9, the data could be backtracked 
from NO. 10 to N0.1, the origin of the last level 2 circulation vector. Node N0.1 could 
then decide to follow R2.4- to N1.4, forwarding the data to node N0.8 where it could 
reach node N0.6 either way around ring Rl .4 (Rl .4- shown as selected). 

5 

[0065] The above capabilities would, for example, find application in advanced 

disruption tolerant networking, as they in effect allow the data to repeatedly retrace its . 
steps back towards the source until a node is able to successfully find an operational path 
to the destination. 

10 

[0066] Instead of backtracking, it is possible to also simply discard all of the 

circulation vector information and restart the routing process as if the data had originated 
at the node where it became halted. 

1 5 [0067] All of the above approaches differ from traditional routing solutions, 

which would have to wait for the network to re-converge on a set of new paths before 
data could be successfully routed around the failure. 

[0068] In a second example scenario, we now consider how the recursive 

20 abstraction of the network into a series of logical levels can be used to enable the 

advertisement of address and performance information across the network, and then the 
subsequent routing of data across the network based solely on the destination address 
information of the control data within for example Ethernet frames or IP packets. 

25 [0069] We first consider, for this second scenario, how the topology of the 

network can be discovered. Existing methods, such as those used by link-state routing 
protocols, are used to communicate the topology across the network enabling each node 
to construct a complete view of the physical topology of the network as shown in Figure 
4. 

30 
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[0070] The logical network abridgement process can then be used to establish the 

recursive abstraction of the physical network into a series of logical levels as shown in 
Figure 6. Each node, as illustrated in Figure 3, independently performs the logical 
network abridgement process on the basis of the network information communicated to it 
5 by implementing software stored in its program memory 41 to populate the routing table 
38. Similarly, software stored in the memory 41 may be used to update the routing table 
38 when new information regarding the network e.g. new addresses or damaged links are 
communicated to the node. 

1 0 [0071] We next consider the advertising of address and performance information 

from a node in level 1 . As an example, take the ring Rl . 1 of level 1 of the network, as 
shown in Figure 15. Using node N0.1 as an example, Figure 15 shows how 
advertisements from one node are circulated around logical ring Rl.l to the other nodes 
in that ring. 

15 

[0072] Each node on the logical ring Rl.l advertises any hosts (not shown) 

attached to it by sending announcement control messages around the ring Rl.l. The 
control messages themselves are addressed to their adjacent neighbours using either a 
specific address for that neighbour that was identified during the topology discovery or 

20 by using a generic address that all nodes will accept. The messages are routed 

appropriately around the network by including within them a circulation vector (CV). 
The messages are associated with a particular ring by using the CV to identify the ring 
(Rl.l in the example). The announcement messages are circulated around the logical 
ring, maintaining either a positive or negative direction of rotation by using the CV to 

25 identify the direction of rotation (Rl . 1+ or Rl . 1 - in the example). To prevent infinite 
looping of the announcement messages the destination of the circulation vector is set 
equal to the originating node so that they can be recognised upon their return to the 
originating node and dropped (R1.1+ to N0.1 or Rl.l- to N0.1 in the example). 
Announcement messages should be sent in both a positive and negative rotation, so that 

30 the system is able to withstand any single link or node failures. The level 1 source of the 
advertisements is also identified within the announcement message to assist with the 
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subsequent routing of data (NQ.l in the example). When announcement messages are 
received, they are used to update the routing tables with the destination addresses that are 
reachable by that route (in this case Nl.l) and the associated diverse paths on the 
underlying closed ring (in this case Rl .1) to the destination node for the route. 

5 

[0073] We now consider the next higher logical level, referred to as level 2, by 

reference to Figure 16 which shows the nodes associated with the level 1 rings R 1.1 to 
R1.5 in white and the level 2 abstraction in bold. The logical network abridgement 
process described earlier details the method of abstraction. The resulting 4 rings in dotted 
1 0 grey R2. 1 , R2.2, R2.3, and R2.4 provide the level 2 abstraction of the network. Thus it 
can be seen that the architecture is recursive, because now the same processes are applied 
again (see below), although they must be developed further to allow for the progressive 
abstraction of the network. 



1 5 [0074] We next consider the advertising of address and performance information 

from a node in level 2. As an example, take the ring R2.1 of the level 2 network, as 
shown in Figure 17. Using node Nl.l as an example, Figure 17 shows how 
advertisements from one node are circulated around logical ring R2.1 to the other nodes 
in that ring. 

20 

[0075] Each node on the logical ring R2.1 advertises any hosts attached to its 

associated ring by sending announcement control messages around the ring R2.1. The 
messages are associated with ring R2. 1 by a CV set equal to R2. 1 . The announcement 
messages are circulated around the logical ring, maintaining either a positive (shown) or 

25 negative (not shown) direction of rotation. To prevent infinite looping of the 

announcement messages the destination of the circulation vector is set equal to the 
originating node Nl.l in the example in Figure 17 - so that they can be recognised upon 
their return to the originating node and dropped. Announcement messages should be sent 
in both a positive and negative rotation around ring R2.1 so that the system is able to 

30 withstand any single link or node failures at that level. 
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[0076] At logical level 2, the link between the level 2 nodes is by way of shared 

nodes in level 1 . To this end, the shared nodes act as level 2 gateways. At logical level 2 5 
the level 2 nodes are level 1 rings. Delivery of information to a level 2 node is therefore 
achieved by circulating it into the associated level 1 ring using the mechanism described 
5 above. As a matter of course this will distribute the information to those nodes in the ring 
that are acting as the level 2 gateway to the next level 1 neighbouring ring, and so the 
process repeats itself. 

[0077] Figure 1 8 shows the level 2 flow of advertisements around ring R2. 1 in a 

1 0 positive direction of rotation from ring Rl . 1 into ring Rl .2 using gateways nodes N0.2 
and NO. 10. Figure 1 8 shows the gateways then circulating the level 2 information around 
the level 1 ring RL2 in a positive direction of rotation. The level 1 proxy source of the 
advertisements is now identified as being the appropriate gateway node NO. 2 or NO. 10. 
To enable the advertisements to follow the level 2 ring R2.1, in the example shown, 

1 5 without looping they have a CV of R2. 1+ to Nl . 1 . To enable the advertisements to 
circulate around R1.2, in the example shown, without looping they also have a CV of 
Rl .2+ to Nl . 1 . By the gateways using the originating level 2 node as the CV 
destination, which is common to both gateways, it will provide an automatic drop and 
substitute action - minimising the flow of address advertisements to a single pair of 

20 rotations (as they correctly drop their partner gateway advertisements assuming that they 
are their own). Announcement messages should be sent in both a positive and negative 
rotation on ring R1.2 so that the system is able to withstand any single link or node 
failures. When announcement messages are received, they are used to update the routing 
tables with the destination addresses that are reachable by that route (in this case N2.1) 

25 and the associated diverse paths on the underlying closed rings (in this case R2.1 and 
below that R1.2) to the destination node. The proxy source addresses can be used to 
identify in the routing table the next level 1 destination for the route. 

[0078] Figure 19 shows the subsequent flow of level 2 advertisements around 

30 R2. 1 in a positive direction from ring Rl .2 into ring Rl .5 using gateway nodes NO. 1 0 and 
N0.1 1. Figure 19 shows the gateways circulating the level 2 information around the level 
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1 ring Rl .5 in a negative direction of rotation. The level 1 proxy source of the 
advertisements is now identified as being the appropriate gateway node NO. 10 or NO. 1 1 . 
To enable the advertisements to continue to follow the level 2 ring R2.1, in the example 
shown, without looping they retain the CV of R2.1+ to Nl.l. To enable the 
5 advertisements to now circulate around Rl .5, in the example shown, without looping they 
also have a CV of RL5- to N1.2. Announcement messages should be sent in both a 
positive and negative rotation on Rl .5 so that the system is able to withstand any single 
link or node failures. When announcement messages are received, they are used to 
update the routing tables with the destination addresses that are reachable by that route 
1 0 (in this case N2. 1) and the associated diverse paths on the underlying closed rings (in this 
case R2.1 and below that Rl .5) to the destination node. The proxy source addresses can 
be used to identify in the routing table the next level 1 destination for the route. 

[0079] Figure 20 shows the level 2 advertisements being dropped on their return 

1 5 from ring Rl .5 to ring Rl . 1 by the gateways in nodes NO. 9 and NO. 1 0. These gateways 
determine from the level 2 CV destination that this level 2 information originated from 
Rl.l and so must be dropped on its return. 

[0080] We now consider the next higher logical level, referred to as level 3, by 

20 reference to Figure 21, which shows, the nodes associated with the level 1 and level 2 
rings in white and the level 3 abstraction in bold. The logical network abridgement 
process described earlier details the method of abstraction. The resulting ring in dotted 
grey R3.1 provides the level 3 abstraction of the network. Thus it can be seen that the 
architecture is recursive, because now the same processes are applied again (see below), 
25 although they must be developed further to allow for the progressive abstraction of the 
network. 

[0081] We next consider the advertising of address and performance information 

from a node in level 3. Considering ring R3.1, as shown in Figure 22 and using node 
30 N2.1 as an example, Figure 22 shows how advertisements from one node are circulated 
around logical ring R3.1 to the other nodes in that ring. 
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[0082] Each node on the logical ring R3. 1 advertises any hosts attached to its 

associated ring of rings by sending announcement control messages around the ring R3.1 . 
The messages are associated with ring R3.1 by a CV set equal to R3.1. The 
5 announcement messages are circulated around the logical ring maintaining either a 
positive (shown) or negative (not shown) direction of rotation. To prevent infinite 
looping of the announcement messages the destination of the circulation vector is set 
equal to the originating node N2.1 in the example in figure 22 - so that they can be 
recognised upon their return to the originating node and dropped. Announcement 
1 0 messages should be sent in both a positive and negative rotation around ring R3. 1 so that 
the system is able to withstand any single link or node failure at that level. 

[0083] At logical level 3, the link between the level 3 nodes is by way of shared 

rings in level 2. To this end, the shared rings act as level 3 gateways. At logical level 3 ? 
1 5 the level 3 nodes are level 2 rings. Delivery of information to a level 3 node is therefore 
achieved by circulating it into the associated level 2 ring using the mechanism described 
above. As a matter of course this will distribute the information to those nodes in the ring 
that are acting as the level 3 gateway to the next level 2 neighbouring ring and so the 
process repeats itself. 

20 

[0084] Figure 23 shows the level 3 flow of advertisements around ring R3.1 in a 

positive direction of rotation from ring R2.1 into ring R2.2 using gateway rings RL2 and 
R1.5. Figure 23 shows the gateways then circulating the level 3 information around the 
level 2 ring R2.2 in a positive direction of rotation from Rl .2 to Rl .3. In this example, 

25 the other flows around the level 2 ring R2.2 are not of interest as in both cases the 

information is dropped as it has returned to its originating level 2 node N2. 1 . To enable 
the advertisements to now circulate R2.2, in the example shown, without looping they 
also have a CV of R2.2+ to N2. 1 . Announcement messages should be sent in both a 
positive and negative rotation around ring R2.2 so that the system is able to withstand any 

30 single link or node failure at that level. 
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[0085] Figure 24 shows the level 2 flow of the advertisements around ring R2.2 in 

a positive direction of rotation from ring Rl .2 into ring Rl .3 using gateway nodes N0.4 
and NO. 1 1 in gateway ring Rl .2. Figure 24 shows the gateways then circulating the level 
3 information, by circulating the level 2 information around the level 1 ring Rl .3 in a 
5 positive direction of rotation. The level 1 proxy source of the advertisements is now 
identified as from the appropriate gateway node N0.4 or NO. 1 1 . To enable the 
advertisements to circulate around R1.3, in the example shown, without looping they also 
have a CV of R1.3+ to N1.2. Announcement messages should be sent in both a positive 
and negative rotation around ring R1.3 so that the system is able to withstand any single 
1 0 link or node failure. When announcement messages are received, they are used to update 
the routing tables with the destination addresses that are reachable by that route (in this 
case N3.1) and the associated diverse paths on the underlying closed rings (in this case 
R3.1, R2.2 and below that Rl .3) to the destination node. The proxy source addresses can 
be used to identify in the routing table the next level 1 destination for the route. 

15 

[0086] Figure 25 shows the subsequent flow of level 3 advertisements around 

R3.1 in a positive direction from ring R2.2 into ring R2.3 using gateway rings R1.3 and 
R1.5. Figure 25 shows the gateways circulating the level 3 information around the level 2 
ring R2.3 in a positive direction of rotation from Rl .3 to Rl .4. In this example, the other 

20 flows around the level 2 ring R2.3 are not of interest as in both cases the information is 
dropped as it has returned to its originating level 2 node N2.2. To enable the 
advertisements to now circulate R2. 3, in the example shown, without looping they also 
have a CV of R2.3+ to N2.2. Announcement messages should be sent in both a positive 
and negative rotation around ring R2.3 so that the system is able to withstand any single 

25 link or node failure at that level. 

[0087] Figure 26 shows the level 2 flow of the advertisements around ring R2.3 in 

a positive direction of rotation from ring Rl .3 into ring Rl .4 using gateway nodes N0.6 
and NO. 12 in gateway ring Rl .3. Figure 26 shows the gateways then circulating the level 
30 3 information, by circulating the level 2 information around the level 1 ring Rl .4 in a 
positive direction of rotation. The level 1 proxy source of the advertisements is now 
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identified as from the appropriate gateway node NO. 6 or NO. 12. To enable the 
advertisements to circulate around R1.4, in the example shown, without looping they also 
have a CV of Rl .4+ to Nl .3. Announcement messages should be sent in both a positive 
and negative rotation around ring R 1.4 so that the system is able to withstand any single 
5 link or node failure. When announcement messages are received, they are used to update 
the routing tables with the destination addresses that are reachable by that route (in this 
case N3.1) and the associated diverse paths on the underlying closed rings (in this case 
R3.1, R2.3 and below that R1.4) to the destination node. The proxy source addresses can 
be used to identify in the routing table the next level 1 destination for the route. 

10 

[0088] The subsequent flow of level 3 advertisements around R3. 1 in a positive 

direction from ring R2.3 into ring R2.4 is achieved in an equivalent manner. Finally the 
flow of level 3 advertisements around R3.1 in a positive direction comes to a halt as the 
level 3 advertisements are dropped when they return to the originating level 3 node N2.1 . 

15 

[0089] We now consider the next higher logical level, referred to as level 4, by 

reference to Figure 27, which shows, the level 1 ,2 and 3 nodes in white and the level 4 
abstraction in bold as a single logical node N3.1. As level 4 forms a loop-free higher- 
level logical topology, the recursive abstractions of the network is complete. There is 
20 now deterministic routing across the whole network. For further clarity the stages of the 
recursive abstraction of the network into a series of logical levels is shown in Figure 6. 

[0090] We next consider the example of data being routed across a network 

without the addition of any circulation vector information as in the first scenario. The 
25 lack of any explicit circulation information on each packet of data creates potential for 
ambiguity and therefore may place restrictions on the types of network that it can be used 
on. 

[0091] Reference is now made to Figure 28, which illustrates an example of data 

30 being routed across the network following completion of the advertising process 
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described above. The source of the data is a host machine attached to node NO. 2 and the 
destination is a host machine with address xyz attached to node N0.6. 

[0092] Each node will have learnt of many possible paths for each route in the 

5 network based on the advertisements it has received at many different levels. Each 
possible path can be considered to have a level that corresponds to the level of the 
advertisement that defined it. To avoid routing loops when using these multiple paths 
across the network, each node must follow three general rules: 

1 . When data is forwarded on an existing ring, the direction of circulation must be 
1 0 maintained at all levels. 

2. Data must not be swapped between rings at the same level. 

3. When paths on multiple levels are available, data must be forwarded on a path of the 
lowest level that is operational. 

Without explicit circulation vector information it is not possible to 
1 5 unambiguously follow rules 1 & 2. However, it is possible to form a good approximation 
to rules 1 & 2 by these alternative rules which do not require explicit circulation vector 
information: 

• Do not forward data back out over the incoming port. 

• Use the incoming port to determine the one or more implicit level 1 circulation 

20 vectors that the data is following and use this information when considering rules 1 & 
2 above. 

[0093] Without explicit circulation vector information it is not possible to reliably 

take advantage of level 2 and above performance information. However, by using the 
25 implicit level 1 circulation vector information of the incoming port it is possible to take 
advantage of level 1 performance information. 

[0094] Although there are limitations from using data without explicit circulation 

vector information being added (as in the first scenario), this approach to routing is useful 
30 for many network scenarios and will be described in the example below. 
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[0095] We next consider Figure 29. The host attached to N0.2 has sent the data 

to N0.2 for forwarding to destination xyz. Node 0.2 is aware of destination xyz at the 
lowest level from advertisements received at level 2 which were delivered by level 2 
rings R2.2 and R2.4 - corresponding to diverse paths associated with a route based on 
5 N3.1. These level 2 advertisements would have been delivered to node NO. 2 by level 1 
rings Rl.l and R1.2. The level 2 advertisements would have therefore had level 1 proxy 
sources of N0.8, N0.9 & NO. 10 (from R2.4) and N0.4, NO. 10 & NO. 11 (from R2.2) - . 
corresponding to the gateway nodes associated with these level 2 paths. Node N0.2 is 
aware of how to reach these level 2 gateway nodes from their advertisements on the level 

1 0 1 rings Rl .1 and Rl .2 to which N0.2 is connected. Level 2 gateway nodes N0.9 and 

N0.1 1 can be ruled out as they can only be reached by passing through the other level 2 
gateway nodes. Based on level 1 performance information, node NO. 2 decides to forward 
the data to the level 2 gateway node NO. 10, by circulating it in a positive direction around 
level 1 ring Rl.l. In effect node N0.2 has chosen to send the data at level 2 to N1.5. 

1 5 However, without explicit circulation vector information it is ambiguous as to whether it 
is following R2.4 in a positive direction or R2.2 in a negative direction. 

[0096] We next consider Figure 30. When the data gets to node NO. 10, it makes 

its own independent routing decision. Node 0.2 is aware of destination xyz at the lowest 

20 level from advertisements also received at level 2 which were also delivered by level 2 
rings R2.2 and R2.4 - corresponding to diverse paths associated with a route based on 
N3.1. These level 2 advertisements would have been delivered to node NO. 10 by level 1 
rings Rl.l, R1.2 and R1.5. The level 2 advertisements would have therefore had level 1 
proxy sources of N0.8, N0.9 & N0.12 (from R2.4) and N0.4, N0.1 1 & N0.12 (from R2.2) 

25 - corresponding to the gateway nodes associated with these level 2 paths. Node NO. 10 is 
aware of how to reach these level 2 gateway nodes from their advertisements on the level 
1 rings Rl.l, R1.2 and R1.5 to which node NO. 10 is connected. However, NO. 10 is 
constrained by the rules that it must not forward the data back out over the incoming 
interface and that it must maintain the direction of level 1 circulation, if necessary, as 

30 implied by the incoming interface. From the incoming interface it knows that it came in 
on either R1.1+ or R1.2-. Therefore it cannot forward the data on Rl.l - to NO. 8 or on 
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Rl .2+ to N0.4. It also knows it cannot reach NO. 1 2 without going through N0.9 or NO. 1 1 . 
This leaves NO. 1 0 with two paths N 1 . 1 + to N0.9 or N 1 .2- to NO. 1 1 . Based on level 1 
performance information, node NO. 10 decides to forward the data to the level 2 gateway 
node N0.9, by circulating it in a positive direction around level 1 ring Rl . 1 . 

5 

[0097] We next consider Figure 3 1 . When the data gets to node N0.9, it makes its 

own independent routing decision. Node N0.9 is aware of destination xyz at the lowest 
level from advertisements received at level 1 from node N0.6 both ways around ring R1.4 
- corresponding to diverse paths associated with a route based on Nl .4. From the 

1 0 incoming interface, it knows that the data arrived on either Rl .1+ or Rl .5-. As the 

outgoing level 1 ring is distinct from the incoming level 1 rings it places no restriction on 
the use of Rl .4 to reach NO. 6. This is consistent with a transition from a level 2 path to a 
level 1 path, which is what is happening although N0.9 does not know this explicitly. 
Based on level 1 performance information node N0.9 decides to forward the data to node 

1 5 N0.6 by circulating it in a positive rotation around ring Rl .4. 

[0098] We next consider Figure 32. When the data gets to node NO. 12, it makes 

its own independent routing decision. Node NO. 12 is aware of destination xyz at the 
lowest level from advertisements received at level 1 from node NO. 6 both ways around 

20 rings Rl .4 and Rl .3 - corresponding to diverse paths associated with a route based on 
N2.3. From the incoming interface, it knows that the data arrived on either Rl .4+ or 
Rl .5-. As the outgoing level 1 paths include an option for Rl .4+ this must be selected to 
both maintain the direction of circulation and avoid a same level ring swap - both of 
which can lead to routing loops. Node NO. 12 therefore decides to forward the data to 

25 node NO. 6 by circulating it in a positive rotation around ring Rl .4. 

[0099] When the data arrives at node NO. 6, it recognises that it is the destination 

for the locally attached host and delivers the data. 

30 [0100] Reference is now made to Figures 33 and 34 which illustrate how the 

present invention may be applied to the more general data communication network 
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shown in Figure 1 . As already mentioned, the network shown in Figure 1 can be 
redefined as a number of level 1 closed rings Rl.l to R1.5 and tree Tl.l, as shown in 
Figure 2. Next tree structure Tl.l is collapsed into the node N0.3, and each of the closed 
rings at level 1 may now be defined as a node in a higher logical level 2 of the network, 
5 shown in Figure 33. 

[0101] Thus, in Figure 33, logical node NL1 corresponds to ring Rl.l, logical 

node Nl .2 corresponds to ring Rl .2, logical node Nl .3 corresponds to ring Rl .3, logical 
node Nl .4 corresponds to ring Rl A and logical node Nl .5 corresponds to ring Rl .5. 

10 

[0102] It can be seen from Figure 33 that the level 2 nodes Nl .2, Nl .4 and Nl .5 

now form a tree structure at level 2 with node Nl .5 at the remote end of the tree. This tree 
is thus collapsed into the node N1.2. 

15 [0103] The three logical nodes Nl.l, N1.2 and N1.3 in Figure 33 themselves form 

a higher level closed ring R2. 1 . Figure 34 shows how the ring R2. 1 can itself be defined 
as an equivalent node N2. 1 in a next higher logical level 3 of the network. The node 
N2.1 now forms a loop-free higher-level topology at level 3. 

20 [0104] Thus, at the highest level shown in Figure 34, the whole route through the 

network is now in the form of a loop-free higher-level topology that provides 
deterministic routing across the network. 

[0105] It will be clear from the above explanations that any planar network, 

25 which can be decomposed into closed rings, can be treated in this way, by using recursion 
to create ever-higher levels of abstraction until a loop-free higher-level topology is 
achieved. The process for abstracting a non-planar network is described below. 

[0106] The process of recursive abstraction into a series of logical levels may also 

30 be optionally terminated at any level greater than the first level and less than the loop-free 
higher-level topology. In this instance, the topology at the terminating level, which will 
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correspond to a graph with at least one ring, can be decimated, either arbitrarily, or using 
any suitable algorithm until it is loop-free. Such an approach will restrict the number of 
diverse paths through the network to a number, smaller than the physical topology 
permits. However, this restricted set of paths can reduce the complexity of the 
5 implementation by limiting the degree of path diversity supported. 

[0107] We will now consider how the early termination of the abstraction process 

may, for example, be used in order to introduce a hierarchy into the network architecture. 

1 0 [0108] If two parts of the network interact with each other as peers, then the two 

parts of the network are said to have a flat relationship with respect to each other. Such a 
flat relationship creates the maximum opportunity for path diversity and performance 
management but requires that both parts be of a similar sophisticated capability. 

1 5 [0109] By contrast, if two parts of the network have a hierarchical relationship to 

one another, then one part will be defined as being superior in the hierarchy than the 
other, and the nodes in the superior part will play a full part in the network, but the nodes, 
other than the common nodes, in the subordinate part will be able to play a simplified 
role in the network and will have a slave-master interaction with the superior part of the 

20 network. 

[0110] Reference is now made to Figure 35 to see how the situation is simplified 

by imposing a hierarchical relationship between ring Rl .5 as the superior part and rings 
Rl.l, R1.2, R1.3 and R1.4 as the subordinate part. Each of the level 1 rings is still 

25 replaced by a single logical node in level 2. For example, level 1 ring Rl is still replaced 
by logical level 2 node Nl.l. However, the imposed hierarchical relationship means that 
the level 2 nodes Nl . 1 , Nl .2, Nl .3, and Nl .4 are only connected to Nl .5, which 
represents the superior part of the network, and are no longer connected to their level 2 
neighbours which represent other subordinate parts of the network. The imposition of 

30 hierarchy is an example of a strategy to guide the early termination of the abstraction 
process resulting in this example with tree T2.1 at level 2. 
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[0111] Reference is now made to Figures 36 to 41 which illustrate the 

incorporation of non-planar links into the architecture by first considering a general 
algorithm and then illustrating this by an example. 

5 

[0112] From a given arbitrary network we construct a maximal planar sub- 

network, by which we mean a network containing all the nodes as the original one but 
with some set of links (hereafter called the set of non-planar links) excluded from it. 
Addition of any link from this set to the maximal planar sub-network makes it non- 
1 0 planar. This set can be chosen to satisfy some additional criteria (e.g. to minimise the 
number of non-planar links, to minimise the overall cost of non-planar links if costs are 
assigned to links, combination of these two criteria, etc.). 

[0113] Every non-planar link is assigned a number (LI, L2, etc). Each of two 

1 5 nodes (say NO.x and NO.y) attached to a non-planar link (say Lz) are assigned point of 
connection numbers (PCz.x and PCz.y) which are carried throughout. By definition non- 
planar links cannot be embedded into the plane without crossing but not intersecting 
another link. 

20 [0114] The following steps are then repeated recursively until all non-planar links 

are embedded: When going to the next level, the point of connection numbers are 
transferred to all rings that contain the node with the corresponding number. Then, at that 
next level, we look for all possible planar connections for the non-planar links. If there 
are conflicting possible planar connections (i.e. they cross) the conflict is resolved 

25 according to some criteria (e.g. by keeping the numbers of possible connections for 
different non-planar links according to their assigned costs). If at least one possible 
connection of a non-planar link has been embedded then, the point of connection 
numbers for that link is not carried to the next level. Connections that have been 
embedded carry implicitly point of connection numbers for the purpose of routing. Point 

30 of connection numbers of non-planar links that have not been embedded are carried to the 
next level. 



-32- 



U.S. Express Mail No. EQ 139 078 383 US 
Substitute Specification as Amended 



[0115] Consider now the representative network from Figure 4 with one added 

non-planar link LI from node N0.3 to node N0.1 1 (see Figure 36). The original network 
will serve as a maximal planar sub-network to which we add point of connection numbers 
5 (PCI. 3 and PC1.1 1) for non-planar link LI (see Figure 37). Going to the second level we 
transfer the point of connection number PC 1.3 to node Nl .2 and the point of connection 
number PC1.1 1 to nodes N1.3, N1.4, N1.5 (Figure 38). There are 3 possible planar 
connections shown as dashed lines on Figure 39, and they all can be implemented 
simultaneously. These possible planar connections are converted into links on level 2 but 
1 0 they carry point of connection numbers for the purpose of routing. 

[0116] After all non-planar links have been implemented we can continue the 

procedure of recursive abstraction into higher levels as usual. This is illustrated by 
Figures 40 and 41. 

15 

[0117] The insertion of non-planar links therefore introduces new routes and 

increases diversity into the routing table from the logical level that it is inserted. 

[0118] With the present invention a method of generating novel routing tables for 

20 a data communication network is provided. The method enables deterministic routing to 
be achieved whilst providing a rich set of diverse paths across the network for each route. 
The method is also particularly suited to both responding quickly to congestion or failure 
at a local part of the network as well as responding progressively to congestion or failure 
in distant parts of the network. 

25 
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